test(testkit): add the testkit library for end-to-end suites - #4724

TL;DR

• Adds testkit, a small library for writing end-to-end tests against a running OpenShell gateway through the Rust SDK. A test is a documented async fn marked #[testkit::test] that runs with plain cargo nextest.
• Tests find the gateway the same way the openshell CLI does, so they run unchanged under e2e/with-docker-gateway.sh and e2e/with-podman-gateway.sh with no extra config.
• Each test gets unique sandbox names, and the sandboxes it creates are cleaned up whether it passes, fails, or panics.
• cx.cli(...) runs the CLI against the same gateway and workspace as the SDK.
• Branch Checks now formats, lints, and unit-tests testkit on all three platforms.
• No suites use it yet and existing e2e lanes are unchanged. This lands the foundation from test(conformance): run conformance suites from test plans in tmachine environments #4629 so the next PR can move suites onto it.

Stack

Part of a stack of four; each PR targets the one below it. The tmachine move (#4726) has merged, so the stack uses its tests/tmachine/ paths.

1. test(testkit): add the testkit library for end-to-end suites #4724: the testkit library.
2. test(sandbox): run a testkit suite in Branch E2E #4822: the first testkit suite, run in Branch E2E against a Docker gateway. The simplest route to testkit tests in CI.
3. feat(runner): add the test runner, plans and tmachine Session API, and run a plan on CI #4773: the test runner, plans, and tmachine Session API, with a plan running on CI.
4. test(suites): port the old suites onto testkit and run them from the plan #4774: ports the old suites onto testkit and adds them to the plan.

Still to come, on top of #4774: moving the release lanes onto plans and deleting the old suites, once the plan has run the ported suites green.

Summary

Adds tests/testkit/, a support library for SDK-driven end-to-end suites, as its own Cargo workspace. This is a small first step to get the crate into main and checked in CI. Nothing uses it yet. The existing suites move onto it in one follow-up migration (#4774, stacked on this PR) that also deletes the old suites. Suites added after that arrive one at a time.

The code comes from @elezar's draft #4629 (head 673928804). This PR takes only the testkit portion, adapted to current main and cut down to what's needed today (see "Deferred and follow-ups" below).

Related Issue

Part of #3954

Changes

• tests/testkit/: the testkit and testkit-macros crates, plus the testkit-ready binary, taken from test(conformance): run conformance suites from test plans in tmachine environments #4629.
  • A test is an async fn marked #[testkit::test] with a required doc comment. Other attributes such as #[ignore] apply to the generated test. #[should_panic] is rejected, because a testkit test panics on any failure.
  • Tests find the gateway the same way the openshell CLI does: OPENSHELL_GATEWAY_ENDPOINT, then OPENSHELL_GATEWAY, then the active gateway. A registered gateway's endpoint and mTLS material come from its CLI config directory. Like the CLI, an endpoint takes its credentials from OPENSHELL_GATEWAY or the registered gateway with that endpoint. cx.cli runs OPENSHELL_BIN against the same gateway and workspace. Each test logs the gateway it used.
  • Cleanup deletes a test's sandboxes concurrently, each with its own timeout.
  • Cargo.lock is re-seeded from the root lockfile, because main's openshell-sdk has more dependencies than when test(conformance): run conformance suites from test plans in tmachine environments #4629 was cut.
• Branch Checks and the rust:lint, rust:format, rust:format:check, and test:rust mise tasks now run fmt, clippy, and nextest for the testkit workspace. This follows test(conformance): run conformance suites from test plans in tmachine environments #4629's cf72f1108, restricted to testkit.
• tests/testkit/README.md covers writing a test, what Context provides, and running against a gateway. TESTING.md, AGENTS.md, and CONTRIBUTING.md point to it.

Deferred and follow-ups

Dropped from #4629's testkit

Each can come back as an optional addition, without changing tests written in the meantime:

• Stable test IDs (id = "t-…"). Until a runner, waivers, or result history need them, the nextest name identifies a test.
• Capabilities and requires = [...]. These belong with the runner, which selects tests per environment. Until then, a suite excludes a test that an environment can't support with a nextest filter or #[ignore]. The first case is test(conformance): run conformance suites from test plans in tmachine environments #4629's stopped_sandbox_keeps_workspace, which needs persistent workspaces.
• The testkit-spec crate, for listing tests without compiling them. It returns with the runner's catalog, if that needs it.
• The target file (OPENSHELL_TEST_TARGET). It held a copy of the CLI's registered gateway plus capabilities, so testkit now reads the CLI's gateway config directly. As a result, step 3 no longer needs test(conformance): run conformance suites from test plans in tmachine environments #4629's openshell_client role change or the runner code that writes target.yaml. The runner only sets OPENSHELL_GATEWAY (or OPENSHELL_GATEWAY_ENDPOINT) and OPENSHELL_BIN.
• Guidance on what a conformance test may assert. It moves to the conformance suites.

Tests ported from #4629 must drop id and requires, because #[testkit::test] now takes no arguments.

Rest of #4629, for later steps

• The runner, plans, environments, waivers, and matrix.
• The tmachine Session library, and the Ansible and config.nix changes.
• The new suites and e2e-suites.yml.
• The lockfile alignment script (test(conformance): run conformance suites from test plans in tmachine environments #4629's a37ebe917). It matters once testkit is built into the Nix test archives. Until then, check-cargo-lockfiles.sh already validates tests/testkit/Cargo.lock.
• Changes to existing e2e lanes.

Follow-ups found while landing this

• Share gateway resolution with the CLI. testkit's gateway.rs repeats the CLI's lookup: the endpoint-to-registered-gateway match, metadata.json auth modes, and the mtls/ files. It also builds the mTLS channel itself, because the SDK leaves mTLS out. The copies can drift apart. When step 3 lands or tmachine needs the same lookup, move it into openshell-bootstrap or an SDK helper so the CLI, testkit, and tmachine share one implementation.
• Other auth modes. testkit supports plaintext and mTLS gateways. A gateway registered with OIDC or edge-token auth is rejected with a clear error. Add support when a lane needs it.
• Small differences from the CLI. Both are documented or harmless for today's wrappers, and both go away if resolution is shared:
  • For an HTTPS URL that no registered gateway uses, testkit connects without a client certificate. The CLI guesses a certificate directory from the hostname instead.
  • testkit doesn't sanitize gateway names when building the mtls/ path, as the CLI does. Gateway names with only letters, digits, ., -, and _ are unaffected.
• Cleaning up more than sandboxes. Cleanup only deletes sandboxes created through cx.sandbox(...). Tests must delete anything else they create, such as providers. This is documented in the README. Extend the cleanup registry when suites create other resources often.
• CI build cache. testkit is its own Cargo workspace, so its build output in tests/testkit/target isn't covered by the Rust cache. This adds roughly 1–2 minutes to each Rust lint and test job. Caching it would add about 2 GB per platform cache. Revisit if suites make the build slower.
• Local runs on Docker Desktop. Sandboxes can't reach a gateway on the Mac's loopback unless Docker Desktop's host networking is on, or the gateway uses supervisor_network_mode = "bridge". e2e/with-docker-gateway.sh sets neither. Linux CI isn't affected. Consider setting bridge mode in the wrapper on macOS when the first testkit lane lands.

Raised in review and left as is

• CLI error messages include the full command line, so a credential passed as an argument would appear in test output. Arguments are already visible in process listings, and suites use throwaway credentials. Hiding values would mean guessing which arguments are secret.
• Some Sandbox helpers have no caller yet (for example config, draft_policy, and set_setting). They came from test(conformance): run conformance suites from test plans in tmachine environments #4629 for the suites in the next PR, so removing them now would only mean adding them back.

Testing

• cargo fmt --manifest-path tests/testkit/Cargo.toml --all -- --check
• cargo clippy --locked --manifest-path tests/testkit/Cargo.toml --all-targets -- -D warnings
• cargo nextest run --locked --config-file .config/nextest.toml --manifest-path tests/testkit/Cargo.toml (1 test: the macro rejects misuse. Gateway resolution, the CLI wrapper, and cleanup are exercised against a real gateway below and by the suites that adopt testkit.)
• tasks/scripts/check-cargo-lockfiles.sh covers tests/testkit/Cargo.lock
• mise run license:check
• Unit tests added/updated (if applicable)
• E2E tests added/updated (if applicable). Not applicable: no suite uses testkit yet.

Against a real gateway

To check that testkit works end to end, I built a throwaway suite crate outside the repo. It depends on this branch's testkit and contains the six sandbox_lifecycle tests from #4629, plus a test that creates three sandboxes, checks that cx.cli(["sandbox", "list"]) shows them, and leaves them for cleanup. I ran it unchanged under e2e/with-docker-gateway.sh on macOS with Docker Desktop. The wrapper's own OPENSHELL_GATEWAY and OPENSHELL_BIN were the only configuration:

e2e/with-docker-gateway.sh sh -c 'testkit-ready 120 && cargo nextest run --manifest-path <suite>/Cargo.toml --test-threads 1'

testkit-ready resolved the wrapper's mTLS gateway, reported it healthy, and confirmed the CLI reaches it. All 7 tests passed in about 15s, and openshell sandbox list showed nothing left behind. Two #[ignore] tests, with the attribute above and below #[testkit::test], were skipped. With only OPENSHELL_GATEWAY_ENDPOINT set, as e2e/rust/e2e-vm.sh does, testkit found the registered gateway's mTLS material, and testkit-ready plus two tests passed.

On Docker Desktop, sandboxes need either Docker Desktop's host networking turned on (Settings → Resources → Network) or supervisor_network_mode = "bridge". Otherwise the supervisor cannot reach a gateway on the Mac's loopback. For this run I set bridge mode with a local, uncommitted change to the wrapper's gateway config.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)